Random Testing For Language Design

Property-based random testing can facilitate formal verification, exposing errors early on in the proving process and guiding users towards correct specifications and implementations. However, effective random testing often requires users to write custom generators for well-distributed random data satisfying complex logical predicates, a task which can be tedious and error prone. In this work, I aim to reduce the cost of property-based testing by making such generators easier to write, read and maintain. I present a domain-specific language, called Luck, in which generators are conveniently expressed by decorating predicates with lightweight annotations to control both the distribution of generated values and the amount of constraint solving that happens before each variable is instantiated. I also aim to increase the applicability of testing to formal verification by bringing advanced random testing techniques to the Coq proof assistant. I describe QuickChick, a QuickCheck clone for Coq, and improve it by incorporating ideas explored in the context of Luck to automatically derive provably correct generators for data constrained by inductive relations. Finally, I evaluate both QuickChick and Luck in a variety of complex case studies from programming languages literature, such as information-flow abstract machines and type systems for lambda calculi

Formalizing Stack Safety as a Security Property

The term stack safety is used to describe a variety of compiler, runtime, and hardware mechanisms for protecting stack memory. Unlike “the heap,” the ISA-level stack does not correspond to a single high-level language concept: different compilers use it in different ways to support procedural and functional abstraction mechanisms from a wide range of languages. This protean nature makes it difficult to nail down what it means to correctly enforce stack safety

Foundational Property-Based Testing

International audienceIntegrating property-based testing with a proof assistant creates an interesting opportunity: reusable or tricky testing code can be formally verified using the proof assistant itself. In this work we introduce a novel methodology for formally verified property-based testing and implement it as a foundational verification framework for QuickChick, a port of QuickCheck to Coq. Our framework enables one to verify that the executable testing code is testing the right Coq property. To make verification tractable, we provide a systematic way for reasoning about the set of outcomes a random data generator can produce with non-zero probability, while abstracting away from the actual probabilities. Our framework is firmly grounded in a fully verified implementation of QuickChick itself, using the same underlying verification methodology. We also apply this methodology to a complex case study on testing an information-flow control abstract machine, demonstrating that our verification methodology is modular and scalable and that it requires minimal changes to existing code

Testing noninterference, quickly

Information-flow control mechanisms are difficult to design and labor intensive to prove correct. To reduce the time wasted on proof attempts doomed to fail due to broken definitions, we advocate modern random testing techniques for finding counterexamples during the design process. We show how to use QuickCheck, a property-based random-testing tool, to guide the design of a simple information-flow abstract machine. We find that both sophisticated strategies for generating well-distributed random programs and readily falsifiable formulations of noninterference properties are critically important. We propose several approaches and evaluate their effectiveness on a collection of injected bugs of varying subtlety. We also present an effective technique for shrinking large counterexamples to minimal, easily comprehensible ones. Taken together, our best methods enable us to quickly and automatically generate simple counterexamples for all these bugs

Deeper Shallow Embeddings

Deep and shallow embeddings are two popular techniques for embedding a language in a host language with complementary strengths and weaknesses. In a deep embedding, embedded constructs are defined as data in the host: this allows for syntax manipulation and facilitates metatheoretic reasoning, but is challenging to implement - especially in the case of dependently typed embedded languages. In a shallow embedding, by contrast, constructs are encoded using features of the host: this makes them quite straightforward to implement, but limits their use in practice. In this paper, we attempt to bridge the gap between the two, by presenting a general technique for extending a shallow embedding of a type theory with a deep embedding of its typing derivations. Such embeddings are almost as straightforward to implement as shallow ones, but come with capabilities traditionally associated with deep ones. We demonstrate these increased capabilities in a number of case studies; including a DSL that only holds affine terms, and a dependently typed core language with computational beta reduction that leverages function extensionality

Automatic WSDL-guided Test Case Generation for PropEr Testing of Web Services

With web services already being key ingredients of modern web systems, automatic and easy-to-use but at the same time powerful and expressive testing frameworks for web services are increasingly important. Our work aims at fully automatic testing of web services: ideally the user only specifies properties that the web service is expected to satisfy, in the form of input-output relations, and the system handles all the rest. In this paper we present in detail the component which lies at the heart of this system: how the WSDL specification of a web service is used to automatically create test case generators that can be fed to PropEr, a property-based testing tool, to create structurally valid random test cases for its operations and check its responses. Although the process is fully automatic, our tool optionally allows the user to easily modify its output to either add semantic information to the generators or write properties that test for more involved functionality of the web services

Automatic WSDL-guided property-based testing of web services

70 σ.Καθώς οι υπηρεσίες διαδικτύου (Web Services) αρχίζουν και αποτελούν ολοένα και βα- σικότερα τμήματα μοντέρνων διαδικτυακών συστημάτων λογισμικού, η ύπαρξη αυτόματων και εύχρηστων αλλά ταυτόχρονα και εκφραστικών προγραμμάτων ελέγχου για υπηρεσίες διαδικτύου καθίσταται όλο και σημαντικότερη. Η διπλωματική αυτή στοχεύει στον πλήρως αυτοματοποιημένο έλεγχο υπηρεσιών διαδικτύου: Ιδανικά, ο χρήστης απλά περιγράφει ιδιό- τητες που οι υπηρεσίες πρέπει να ικανοποιούν, με τη μορφή σχέσεων εισόδου-εξόδου, και το σύστημά μας αναλαμβάνει τα υπόλοιπα. Σε αυτή τη διπλωματική περιγράφουμε αναυτικά όα τα επιμέρους τμήματα του εργαλείου που φτιάξαμε: Πώς οι προδιαγραφές (WSDL) μιας υπηρεσίας διαδικτύου χρησιμοποιούνται για να παραχθούν με αυτόματο τρόπο γεννήτριες συντακτικά ορθών τυχαίων δεδομένων και ιδιότητες, οι οποίες μπορούν να δοθούν στο PropEr, ένα εργαλείο ελέγχου μέσω ιδιοτήτων, ώστε να κληθούν οι μέθοδοι της υπηρεσίας διαδικτύου και να ελεγχθεί η απόκρισή τους. Παρόλο που η διαδικασία είναι πλήρως αυτοματοποιημένη, το εργαλείο δίνει τη δυνατότητα στο χρήστη να αλλάξει το παραγόμενο αρχείο ελέγχου που περιέχει τις ιδιότητες και τις γεννήτριες ώστε να έχει μεγαλύτερο έλεγχο στην όλη διαδικασία και να μπορέσει να ελέγξει πιο στοχευμένα την λειτουργικότητα της υπηρεσίας διαδικτύου.With web services already being key ingredients of modern web systems, automatic and easy-to-use but at the same time powerful and expressive testing frameworks for web services are increasingly important. Our work aims at fully automatic testing of web services: ideally the user only specifies properties that the web service is expected to satisfy, in the form of input-output relations, and the system handles all the rest. In this thesis we present in detail all the components which form this system: how the WSDL specification of a web service is used to automatically create test case generators and properties that can be fed to PropEr, a property-based testing tool, to create structurally valid random test cases for its operations and check its responses. Although the process is fully automatic, our tool optionally allows the user to easily modify its output to either add semantic information to the generators or write properties that test for more involved functionality of the web services.Λεωνίδας Κ. Λαμπρόπουλο

Testing noninterference, quickly

Information-flow control mechanisms are difficult to design and labor intensive to prove correct. To reduce the time wasted on doomed proofs for broken definitions, we advocate modern random testing techniques for finding counterexamples during the design process. We show how to use QuickCheck, a property-based random-testing tool, to guide the design of a simple informationflow abstract machine. We find that both sophisticated strategies for generating well-distributed random programs and readily falsifiable formulations of noninterference properties are critically important. We propose several approaches and evaluate their effectiveness on a collection of injected bugs of varying subtlety. We also present an effective technique for shrinking large counterexamples to minimal, easily comprehensible ones. Taken together, our best methods enable us to quickly and automatically generate simple counterexamples for all these bugs